Examining the Consistency of Evaluations Provided by Three Automatic Speech Recognition Systems

In an EFL setting like Taiwan, it is often difficult to provide individual oral language training due to limited human resources. Recently, several innovative commercial CALL programs (e.g., TeLLmeMore and MyET) claim that they can provide high-quality pronunciation training with their automatic speech recognition technologies. Although these programs can provide scores and feedback to individual learners, many teachers and students are not sure if the evaluations provided by these systems are fair and consistent. In this study, the consistency of evaluations provided by three different ASR programs (TeLLmeMore, MyET, and Microsoft) was examined. Four different groups of college students were asked to interact with these different ASR systems. The subjects read the same 20 sentences to each system and then received scores and feedback from these three systems. The whole interaction processes and subjects’ performance and scores were recorded by Camtasia, a screen and audio capture tool. The audio files of the subjects’ performance were also evaluated by two human raters. Based on the scores assigned by human raters and ASR programs, strong correlations were found between human raters and different ASR scores. Moreover, strong correlations were also found among different ASR systems. These statistical results show that automatic speech recognition technologies can indeed assign consistent scores to students at different proficiency levels. Although at this stage these ASR systems might not be suitable for high-stake tests like TOEFL or college entrance examinations, these tools can be used for pedagogical interventions and low-stake tests like placement tests or diagnostic tests.